Every few years the machine vision world gets a pitch that sounds too good to check: stop tuning thresholds, just tell the system what a defect looks like. This year it’s vision-language models — VLMs — bolted onto inspection stations, and the pitch has gotten genuinely better. Cognex, Keyence, and Landing AI have all shipped tooling that lets you type or speak a description of a flaw and get a working classifier back in minutes instead of weeks. Open-source VLM stacks built on models like CLIP variants or larger multimodal foundation models are showing up in integrator toolkits too. The demos are real. The question is whether they survive contact with your actual line.
My position: VLM-based inspection is a legitimate new tool, not a replacement for your Cognex or Keyence ruleset, and treating it as a wholesale swap-in is how you end up explaining a missed shipment to your plant manager. The right way to think about it is as a second tool for a category of problems your rule-based system was never good at — not a faster way to rebuild the system you already have working.
What VLMs actually change
Classic machine vision — blob analysis, edge detection, pattern matching, and even standard CNN-based classifiers — needs you to define the defect space up front. You collect images, you label them, you tune thresholds or train a model, and the system gets good at catching exactly what you showed it and nothing else. That’s a strength when your defect modes are stable and well understood. It’s a weakness when defects are novel, rare, or hard to describe in geometric terms — the “it just looks wrong” defects that a trained inspector catches by eye but that resist a clean edge-detection rule.
VLMs are trained on huge, general image-text corpora, so they carry a prior understanding of concepts like “scratch,” “discoloration,” “misalignment,” or “flash” without you having to build a labeled dataset from scratch. Zero-shot or few-shot prompting lets an engineer describe a defect in natural language and get a first-pass classifier running same-day. That’s a real capability shift, and it’s why the vendor demos land — a quality engineer who’s spent a career fighting lighting rigs and threshold sliders sees a text box and thinks, finally.
Where it holds up on a real line
The genuine sweet spot is high-mix, low-volume, or early-NPI inspection stations where building a full ruleset or training a CNN for every SKU isn’t worth the engineering time — think contract manufacturers running dozens of part numbers through the same station, or a new product introduction where defect modes aren’t fully characterized yet. VLMs are also useful for anomaly triage: flagging “this looks different from the reference set” for human review, rather than making a final accept/reject call. Used that way, they extend your inspectors instead of replacing your PLC-tied reject logic.
They’re also legitimately good at reading and reasoning across modalities — matching a label to a spec sheet, checking that an assembly step matches a work instruction image, verifying presence/absence of a component described in words rather than a fixed template. That’s a different job than pixel-level surface defect detection, and it’s one classic vision has always struggled to do cheaply.
Where it fails, and fails quietly
The failure modes matter more than the wins, because they don’t always look like failures. A few to watch for:
- Subtle surface defects. Hairline cracks, sub-millimeter porosity, low-contrast scuffs on textured surfaces — the stuff that classic vision handles with tuned lighting and high-resolution line-scan cameras — is exactly where general-purpose VLMs underperform. They’re pattern-matching against broad visual concepts, not calibrated against your material’s specific defect signature. A model can sound confident about a call it’s wrong about, with no threshold value you can adjust to fix it.
- Lighting and fixture drift. Rule-based systems are brittle to lighting changes, but that brittleness is visible — the system fails hard and obviously, and you retune it. VLM outputs can drift more gracefully and less visibly, quietly shifting confidence scores in a way that’s harder to catch in routine SPC review.
- Cycle time. This is the one that kills pilots. A Cognex In-Sight or Keyence CV-X system running a tuned ruleset executes in single-digit to low double-digit milliseconds, synced to a PLC trigger on a line running at takt times measured in fractions of a second. A cloud-hosted or even locally-served VLM inference call, especially one doing genuine multimodal reasoning rather than a lightweight embedding lookup, can run one or two orders of magnitude slower. If your station runs at high line speed, that gap isn’t a tuning problem — it’s an architecture problem.
- Explainability and audit trail. Automotive and medical device quality systems built around IATF 16949 or ISO 13485 expect a defensible, repeatable basis for accept/reject decisions. “The model said so” is a harder story to tell an auditor than a documented threshold on a measured feature.
How to structure a pilot that won’t blow up your line
Don’t run the pilot on your existing inspection station’s live reject decision. Run it in parallel, on the same image stream, in shadow mode — the VLM sees every part your current system inspects, logs its call, but doesn’t gate the line. Compare against your existing ruleset and against human audit for a real production run, not a curated demo set, and do it long enough to span your normal lighting drift, shift changes, and material lot variation. That’s the only way you catch the quiet failure modes instead of the dramatic ones.
Measure latency under real line speed, not bench conditions. If the VLM can’t hit your takt time on your actual hardware — including network round-trip if it’s not fully on-prem — it’s not a production candidate yet, full stop, regardless of how good its accuracy looks in shadow mode.
Scope the pilot to a defect category your current system genuinely struggles with, not the one it already handles well. If you pick a problem your Cognex setup solved five years ago, you’ll get a demo that proves nothing and a vendor relationship that assumes you’re the client. Pick the SKU changeover pain, the novel-defect blind spot, the NPI line without a mature ruleset yet.
Keep ownership of the decision logic. Some platforms let you use a VLM for detection and flag generation while keeping a deterministic reject signal in the PLC logic — that’s a safer integration pattern than letting model output drive the actuator directly, at least until you’ve built up a real track record.
None of this means skip it. The technology is moving fast enough, and the vendor tooling is maturing fast enough, that ignoring it is its own risk. But the plants that get value out of this next wave will be the ones running disciplined shadow-mode pilots against a real production baseline — not the ones who ripped out a working ruleset because a sales engineer typed “scratch” into a text box and it worked once.
This article was written with the assistance of artificial intelligence. While we aim for accuracy, the information may be incomplete, out of date, or incorrect, and should be independently verified before you rely on it for any decision. It is provided for general information only and does not constitute professional advice.
